191 research outputs found
Augmenting transformers with recursively composed multi-grained representations
We present ReCAT, a recursive composition augmented Transformer that is able
to explicitly model hierarchical syntactic structures of raw texts without
relying on gold trees during both learning and inference. Existing research
along this line restricts data to follow a hierarchical tree structure and thus
lacks inter-span communications. To overcome the problem, we propose a novel
contextual inside-outside (CIO) layer that learns contextualized
representations of spans through bottom-up and top-down passes, where a
bottom-up pass forms representations of high-level spans by composing low-level
spans, while a top-down pass combines information inside and outside a span. By
stacking several CIO layers between the embedding layer and the attention
layers in Transformer, the ReCAT model can perform both deep intra-span and
deep inter-span interactions, and thus generate multi-grained representations
fully contextualized with other spans. Moreover, the CIO layers can be jointly
pre-trained with Transformers, making ReCAT enjoy scaling ability, strong
performance, and interpretability at the same time. We conduct experiments on
various sentence-level and span-level tasks. Evaluation results indicate that
ReCAT can significantly outperform vanilla Transformer models on all span-level
tasks and baselines that combine recursive networks with Transformers on
natural language inference tasks. More interestingly, the hierarchical
structures induced by ReCAT exhibit strong consistency with human-annotated
syntactic trees, indicating good interpretability brought by the CIO layers.Comment: preprin
Simple Hardware-Efficient PCFGs with Independent Left and Right Productions
Scaling dense PCFGs to thousands of nonterminals via a low-rank
parameterization of the rule probability tensor has been shown to be beneficial
for unsupervised parsing. However, PCFGs scaled this way still perform poorly
as a language model, and even underperform similarly-sized HMMs. This work
introduces \emph{SimplePCFG}, a simple PCFG formalism with independent left and
right productions. Despite imposing a stronger independence assumption than the
low-rank approach, we find that this formalism scales more effectively both as
a language model and as an unsupervised parser. As an unsupervised parser, our
simple PCFG obtains an average F1 of 65.1 on the English PTB, and as a language
model, it obtains a perplexity of 119.0, outperforming similarly-sized low-rank
PCFGs. We further introduce \emph{FlashInside}, a hardware IO-aware
implementation of the inside algorithm for efficiently scaling simple PCFGs.Comment: Accepted to Findings of EMNLP, 202
Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks
Entity and Relation Extraction (ERE) is an important task in information
extraction. Recent marker-based pipeline models achieve state-of-the-art
performance, but still suffer from the error propagation issue. Also, most of
current ERE models do not take into account higher-order interactions between
multiple entities and relations, while higher-order modeling could be
beneficial.In this work, we propose HyperGraph neural network for ERE
(\hgnn{}), which is built upon the PL-marker (a state-of-the-art marker-based
pipleline model). To alleviate error propagation,we use a high-recall pruner
mechanism to transfer the burden of entity identification and labeling from the
NER module to the joint module of our model. For higher-order modeling, we
build a hypergraph, where nodes are entities (provided by the span pruner) and
relations thereof, and hyperedges encode interactions between two different
relations or between a relation and its associated subject and object entities.
We then run a hypergraph neural network for higher-order inference by applying
message passing over the built hypergraph. Experiments on three widely used
benchmarks (\acef{}, \ace{} and \scierc{}) for ERE task show significant
improvements over the previous state-of-the-art PL-marker.Comment: Accepted to Proceedings of EMNLP, 202
Application of the Variational Mode Decomposition for Power Quality Analysis
Harmonics and interharmonics in power systems distort the grid voltage, deteriorate the quality and stability of the power grid. Therefore, rapid and accurate harmonic separation from the grid voltage is crucial to power system. In this article, a variational mode decomposition-based method is proposed to separate harmonics and interharmonics in the grid voltage. The method decomposes the voltage signal into fundamental, harmonic, interharmonic components through the frequency spectrum. An empirical mode decomposition (EMD) and an ensemble empirical mode decomposition (EEMD) can be combined with the independent component analysis (ICA) to analyze the harmonics and intherharmonics. By comparing EMD-ICA, EEMD-ICA methods, the proposed method has several advantages: (1) a higher correlation coefficient of all the components is found; (2) it requires much less time to accomplish signal separation; (3) amplitude, frequency, and phase angle are all retained by this method. The results obtained from both synthetic and real-life signals demonstrate the good performance of the proposed method
A Comprehensive Survey on Database Management System Fuzzing: Techniques, Taxonomy and Experimental Comparison
Database Management System (DBMS) fuzzing is an automated testing technique
aimed at detecting errors and vulnerabilities in DBMSs by generating, mutating,
and executing test cases. It not only reduces the time and cost of manual
testing but also enhances detection coverage, providing valuable assistance in
developing commercial DBMSs. Existing fuzzing surveys mainly focus on
general-purpose software. However, DBMSs are different from them in terms of
internal structure, input/output, and test objectives, requiring specialized
fuzzing strategies. Therefore, this paper focuses on DBMS fuzzing and provides
a comprehensive review and comparison of the methods in this field. We first
introduce the fundamental concepts. Then, we systematically define a general
fuzzing procedure and decompose and categorize existing methods. Furthermore,
we classify existing methods from the testing objective perspective, covering
various components in DBMSs. For representative works, more detailed
descriptions are provided to analyze their strengths and limitations. To
objectively evaluate the performance of each method, we present an open-source
DBMS fuzzing toolkit, OpenDBFuzz. Based on this toolkit, we conduct a detailed
experimental comparative analysis of existing methods and finally discuss
future research directions.Comment: 34 pages, 22 figure
- …